Crowdsource authoring as a tool for enhancing the quality of competency assessments in healthcare professions

The current Objective Structured Clinical Examination (OSCE) is complex, costly, and difficult to provide high-quality assessments. This pilot study employed a focus group and debugging stage to test the Crowdsource Authoring Assessment Tool (CAAT) for the creation and sharing of assessment tools used in editing and customizing, to match specific users’ needs, and to provide higher-quality checklists. Competency assessment international experts (n = 50) were asked to 1) participate in and experience the CAAT system when editing their own checklist, 2) edit a urinary catheterization checklist using CAAT, and 3) complete a Technology Acceptance Model (TAM) questionnaire consisting of 14 items to evaluate its four domains. The study occurred between October 2018 and May 2019. The median time for developing a new checklist using the CAAT was 65.76 minutes whereas the traditional method required 167.90 minutes. The CAAT system enabled quicker checklist creation and editing regardless of the experience and native language of participants. Participants also expressed the CAAT enhanced checklist development with 96% of them willing to recommend this tool to others. The use of a crowdsource authoring tool as revealed by this study has efficiently reduced the time to almost a third it would take when using the traditional method. In addition, it allows collaborations to partake on a simple platform which also promotes contributions in checklist creation, editing, and rating.

the submission is accepted.Please make sure it is accurate.

Unfunded studies
Enter: The author(s) received no specific funding for this work.Yes -all data are fully available without restriction Tick here if the URLs/accession numbers/DOIs will be available only after acceptance of the manuscript for publication so that we can ensure their inclusion before publication.

Introduction
During the past 5 decades, the focus of the training of medical has shifted dramatically towards competency-based medical education (CBME) [1].Thus, the development of new assessment methodologies for the accurate measurement of competency in an accurate and unbiased form was needed [2].The traditional Objective Structured Clinical Examination (OSCE) was developed in the 1970s as a replacement for the Clinical Examination, which was a required practical competency exam for graduating physicians in the United Kingdom [3].This examination is a 1-hour encounter with a patient, where candidates are requested to obtain a clinical history and conduct a physical examination.After the allotted time, the candidate would then meet with a group of examiners to discuss the case further, approximately 20 minutes.Then, the candidate is required to perform a focused examination of several additional patients and discuss the clinical findings with the examiners.
OSCE has been in place for several decades, however, it attracted a significant amount of criticism, due to lack of reliability in measuring candidates' clinical competence and the effect of no implemented standardization of cases and examiners.Later developments to improve the quality of OSCE in terms of validity and reliability scores were the design of scenario and assessment tool design checklists, more frequently used as interactive digital files [4].The basic procedure for developing OSCE checklists includes three main stages for developing assessment tools to measure clinical skills [5].These stages constitute the traditional method of checklist development which is still used by numerous institutions around the world: 1) Preliminary List of Measurable Steps -the OSCE station authors develop an initial list of all key and measurable steps in the clinical skill to be evaluated in a particular station; 2) Specialist Assessment -experts with particular skills assess the OSCE station's draft checklists and provide feedback and suggestions for improvement; 3) Field Testing -a mock examination using realistic OSCE conditions.
However, the electronic checklist system, a worldwide recent development, can eliminate missing data and decrease post-assessment working loads.Therefore, the quality of future medical education will rely on increased collaboration between institutions providing medical programs [10][11][12].These inter-institutional collaborations could enhance the quality of medicine using joint development of educational programs and activities by combining the resources and capabilities of different medical schools [13].One example is the MedEdPORTAL system (Figure 1).It recognizes author contributions by displaying publication metrics on its website.
The built-in credit system displays specific metrics such as the number of views, number of downloads, and the conversion rate from views to downloads.The system also tracks social media coverage and displays an "attention score" calculated based on these metrics.The system tracks the impact of the shared resource but indirectly acts as a reward for the authors since it is an indicator of quality and relevance.
One of the greatest examples of successful crowd authoring and crowdsourcing not only in the medical field but all other fields is Wikipedia.This is an online encyclopedia that is completely crowd authored; anyone with knowledge of a particular subject can contribute.
Contributions are added to the database after a peer-review process to ensure accuracy.By December 2020, Wikipedia contained more than 6 million crowdsourced articles and has become one of the largest repositories of online information in the world [14,15].Crowdsourcing comes from a less-specific, more public group and its advantages may include improved costs, speed, quality, flexibility, scalability, or diversity.Therefore, this study aims to build a Crowdsource Assessment Authoring Tool (CAAT) which can generate an OSCE checklist easier and more efficiently.

Methods
The online OSCE generator checklist CAAT conceptualized in this study was comprised of three phases.The first phase establishes frameworks for procedure-based OSCE checklists and online platforms (Figure 2).The second phase is aimed at detecting and fixing bugs (e.g. problems with software, programs).The third phase investigates the user's acceptance of the CAAT.The first two phases are designed for practical purposes, and the third phase contains the theoretical investigation.

Consent and ethics statement
Participants were given informed consent about the study and were explained in detail how their participation can help improve collaboration, checklist creation and to save time and resources in the healthcare profession.This study was approved by the Ethics Committee Review Board at Taipei Medical University at Taipei Medical University Hospital.IRB: TMU-JIRB-N201603019.

Participants in the development of the CAAT framework
1.
Medical doctors and educational scholars 2.
Medical doctors must have at least three years of experience participating in the national OSCE certification or developing OSCE checklists.

3.
Educational scholars must have at least three years of experience handling medical education projects or have published three medical education studies in SCI journals.
Three medical doctors and three educational scholars fulfilling the above criteria participated in the study with a total of 6 panels in the focus group meetings (minimum requirement for the number of panels in the group meeting n=5 for the consensus group of different disciplines).

Phase I: Development of the framework for the CAAT
Researchers examined the recent version of the national OSCE for drafting a CAAT framework.During the drafting process, they followed the concept of ontology or mind map in order to identify the elements of the OSCE checklist and to identify their relationships.As clinical skills completion occurs in a step-by-step manner, an important element of OSCE is time series.Thus, the first layer for the CAAT (time) consists of preparation, processing, postmanagement.The second layer is "the key tasks" for each clinical skill.Then, researchers drafted the first focus group framework.
During the focus group meeting, researchers presented the primary concept and framework of the CAAT and guided the group in developing an OSCE checklist.According to the experience exchange and verified the researchers further modified and enhanced the elements used in developing the OSCE checklist.Afterward, the researchers combined their knowledge and skills to extract the concepts/elements for developing an OSCE checklist.This cycle continued until no new information or modifications were produced.Then, researchers turned their confirmed findings to the engineer which implemented the CAAT's IT framework (alpha 1.0 version, Figure 3).

Phase II: CAAT problems detection and fixing
After the engineer built the first version of the CAAT (alpha 1.0), the researchers examined the systematic problems with other invited experts.Then, the study called the second focus group to identify and fix existing problems.The same criteria for expert panels were used as in Phase I: 1) medical doctors, 2) educational scholars and 3) their required qualifications.
Five experts tested the Phase II of CAAT in a limited period (one week).In this phase, the researchers addressed the gap between IT design and user's needs and identified some system problems.The engineer re-designed and adjusted the CAAT according to the recommendations of the panels, and the researchers re-tested the modified CAAT (alpha 1.1 version) using various medical faculty.The adjusting cycle continued until no further suggestions were given and no problems or issues were encountered and the CAAT (beta 1.0 version, Figure 3) was approved for phase III test and survey.

Phase III: The CAAT (beta version) for user test and survey
To further verify the CAAT (beta 1.0 version) can benefit Taiwan and healthcare systems worldwide, a user test and survey for the OSCE checklist development were implemented.The initial sample size quota for this particular survey was 50 experts.Therefore, the study invited 60 experts from 9 different countries within the three affiliated international simulation centers to participate in Phase III development.The invited experts, faculty members, and/or researchers all attended international assessment training courses between 2010 to 2019.
The CAAT user experience survey was performed through Google docs and included background information, the CAAT acceptance scale, and the CAAT impact.To obtain unbiased feedback, this study used an anonymous survey format (See appendix).The CAAT acceptance scale was adopted from a verified scale [16], and consisted of 14 items reflecting four domains of a technology acceptance model.The domains are perceived in terms of user-friendliness, usefulness, attractiveness, and intention of use.The scale for information system user acceptance was verified and the four domains revealed the following: The Cronbach's alpha for the usefulness domain was .90, the alpha for user friendly was .87, the alpha for attractiveness was .86,and the alpha for the behavioral intention was .87(Table 2).
The CAAT acceptance scale involved four additional domains.The usefulness domain consists of items 1 to 4, the user-friendliness consists of items 5 to 8, attractiveness consists of items 9 to 12, and the behavioral intention consists of item 13 to item 14.Aside from the attractiveness domain, the other three domains used a 7-point Likert scale.The attractiveness domain used a 7-point bipolar scale.

Statistical analysis
This study used descriptive statistics (percentage event and mean with SD) for background information and inferential statistics for testing the study's hypothesis.The percentage event represented nationality, gender, and occupation.The SD mean represented experience for clinical skill teaching, checklists used for education purposes, checklist development, and the average time for developing a checklist.
For inferential statistics, an independent sample t-test, Pearson correlation, one-sample t-test, and dependent sample t-test were conducted as per hypotheses.Pearson correlation served the analysis of the correlation between the user acceptance of the CAAT and experience on clinical skill teaching (hypothesis 1), experience for checklists used in educational purposes, and experience for checklist development (hypothesis 2).A one-sample t-test was used for testing the acceptance of the CAAT among the experts (hypothesis 3).Since the questionnaire was based on a 7-point scale, the test value for the one-sample t-test was set at 5. The dependent sample t-test assessed the difference in time for developing a checklist between the CAAT and the traditional OSCE (hypothesis 4).As the p-value was lower than 0.05, the result reached statistical significance.

Overall results of the CAAT acceptance survey
The following feedback was obtained from the experts participating in CAAT development and testing (Tables 2 and 3

Participants' acceptance of CAAT
The results of the one-sample t-test with a test value of 5 showed that the experts agreed CAAT is a useful, easy, and interesting system for generating a checklist for educational purpose (Table 2).More specifically, the results showed that: (i) a user can quickly decide which items to use for checking students' performance in urinary catheter insertion (MD = 0.94, t = 6.80, p < .001); (ii) can easily decide which item to use for checking students' performance in urinary catheter insertion (MD = 1.00, t = 8.25, p < .001);(iii) can develop an accurate checklist to assess students' performance in urinary catheter insertion (MD = 0.92, t = 6.89, p < .001);(iv) CAAT provides information in ways to effectively develop a checklist for urinary catheter insertion (MD = 0.84, t = 5.09, p < .001);(v) interaction with CAAT is efficient and responsive (MD = The mean difference between the two approaches was -148.80 minutes with marginal significance (t = -1.75,p = .093).Participants/experts from around the world perceived that CAAT could change current practices of checklist development (item 12: MD = 0.76, t = 5.07, p < .001;Table 2).

Discussion
This study developed and tested the Crowdsource Authoring Assessment Tool (CAAT), a new tool and collaboration system to faster and more efficiently generate a checklist for assessing the clinical skill performance of learners.Analyzed data collected from 50 recognized international experts in the field of competency assessment revealed that CAAT can significantly reduce time and improve efficiency and ease when generating a checklist to assess a learner's clinical skills (Table 3).
The average time for participating experts was 65.76 minutes when using CAAT, whereas, they spent 167.90 minutes using OSCE, the traditional method currently used worldwide [1][2][3].
According to the study's results, the average time saved by using CAAT is 102.14 minutes (Table 3).In addition, Cronbach's alpha revealed high reliability for usefulness (.90), user-friendliness (.87), attractiveness (.86), and behavioral intention (.87) which are all important features of CAAT (Table 2).To analyze these important features of CAAT in an unbiased and comprehensive manner, experts from around the world were invited to participate, which are both native (30%) and nonnative English speakers (70%).In addition, the experts participating in this study comprised a variety of professionals from physicians to bioengineers (Table 1).It is important to have a diverse group of experts from around the world in order to better assess the efficiency of the tool in terms of its usefulness, ease of use and from a non-English native perspective [10,11].

Figures
Funded studies Enter a statement with the following details: Initials of the authors who received each award • Grant numbers awarded to each author • The full name of each funder • URL of each funder website • Did the sponsors or funders play any role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript?• NO -Include this sentence at the end of your statement: The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.• YES -Specify the role(s) played.• * typeset Competing Interests Use the instructions below to enter a competing interest statement for this submission.On behalf of all authors, disclose any competing interests that could be perceived to bias this work-acknowledging all financial support and any other relevant financial or nonfinancial competing interests.This statement is required for submission and will appear in the published article if the submission is accepted.Please make sure it is accurate and that any funding sources listed in your Funding Information later in the submission form are also declared in your Financial Disclosure statement.View published research articles from PLOS ONE for specific examples.Authors declare no competing interests Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation NO authors have competing interests Enter: The authors have declared that no competing interests exist.Authors with competing interests Enter competing interest details beginning with this statement: I have read the journal's policy and the authors of this manuscript have the following competing interests: [insert competing interests here] * typeset Ethics Statement Enter an ethics statement for this submission.This statement is required if the study involved: /A" if the submission does not require an ethics statement.General guidance is provided below.Consult the submission guidelines for detailed instructions.Make sure that all information entered here is included in the Methods section of the manuscript.IRB: TMU-JIRB-N201603019. Taipei Medical University Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation Format for specific study types Human Subject Research (involving human participants and/or tissue) Give the name of the institutional review board or ethics committee that approved the study • Include the approval number and/or a statement indicating approval of this research • Indicate the form of consent obtained (written/oral) or the reason that consent was not obtained (e.g. the data were analyzed anonymously) • Animal Research (involving vertebrate animals, embryos or tissues) Provide the name of the Institutional Animal Care and Use Committee (IACUC) or other relevant ethics board that reviewed the study protocol, and indicate whether they approved this research or granted a formal waiver of ethical approval • Include an approval number if one was obtained • If the study involved non-human primates, add additional details about animal welfare and steps taken to ameliorate suffering • If anesthesia, euthanasia, or any kind of animal sacrifice is part of the study, include briefly which substances and/or methods were applied • Field Research Include the following details if this study involves the collection of plant, animal, or other materials from a natural setting: Field permit number • Name of the institution or relevant body that granted permission • Data Availability Authors are required to make all data underlying the findings described fully available, without restriction, and from the time of publication.PLOS allows rare exceptions to address legal and ethical concerns.See the PLOS Data Policy and FAQ for detailed information.
): (i) I can quickly decide the items I want to use for checking students' performance in urinary catheter insertion (M = 5.94, SD = 0.98); (ii) I can easily decide the items I want to use for checking students' performance in urinary catheter insertion (M = 6.00,SD = 0.86); (iii) I can develop an accurate checklist to assess students' performance in urinary catheter insertion (M = 5.92, SD = 0.94); (iv) the CAAT provides me with information that allows me to develop an effective checklist for urinary catheter insertion (M = 5.84, SD = 1.17); (v) the interaction with CAAT is clear and understandable (M = 6.06,SD = 0.89); (vi) the interaction with CAAT does not increase the workload when creating a checklist (M = 6.22,SD = 0.91); (vii) I find CAAT easy to use (M = 6.16,SD = 0.74); (viii) it is easy to get CAAT to do what I need (M = 6.02,SD = 0.98); (ix) I feel CAAT is enjoyable/awful (M = 6.02,SD = 0.89); (x) I feel CAAT is exciting/dull (M = 5.82, SD = 1.08); (xi) I feel CAAT is pleasant/unpleasant (M = 5.80, SD = 0.97); (xii) I feel CAAT is interesting/boring (M = 5.76, SD = 1.00); (xiii) I intend to revisit CAAT in the future (M = 6.08,SD = 1.01); and (xiv) I will use CAAT next time I need to generate OSCE checklist (M = 5.88, SD = 1.22).

Fig. 1
Fig. 1.The MedPortal page for social media and the downloading of geographic information

.Fig. 2 .
Fig. 1.The MedPortal page for social media and the downloading of geographic information

Fig. 3 .
Fig. 3.The front page of the CAAT system version 1.0.
. a English speaker (n = 15) and non-English speaker (n = 35); b senior (experience on developing checklist more than 4 years, n = 25) and junior (experience on developing checklist within 4 years, n = 25).
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation (xiii) user intends to use CAAT in the future (MD = 1.08, t = 7.58, p < .001);and (xiv) user will likely use CAAT for generating the next OSCE checklist (MD = 0.88, t = 5.09, p < .001).CAAT in about 90.40 minutes, and with the traditional method in 239.20 minutes.
96.60 minutes.The mean difference between the two approaches was -55.48 minutes (t = -3.76,p = .001).However, among the senior experts, the results demonstrated that they generated a new checklist with

Table 1 .
Demographics and characteristics of participants

Table 3 .
Difference in developing a checklist between the CAAT and traditional method.
Paired t-test.M, mean; SD, standard deviation